refactor: unify LinkNavigator truth-link handling - #2780
Conversation
There was a problem hiding this comment.
Pull request overview
This PR refactors truth-link handling by consolidating podio::LinkNavigator patterns behind a shared helper (LinkTruthUtils.h) and migrates several PID / far-detector / calorimetry paths from association-based inputs to link-based inputs, updating plugin wiring and tests accordingly.
Changes:
- Introduce
src/algorithms/interfaces/LinkTruthUtils.hto centralize per-eventLinkNavigatorsetup, weighted link+association emission, and primary-particle extraction. - Migrate multiple algorithms/factories and global/detector wiring from
*AssociationCollectioninputs to*LinkCollectioninputs. - Update
pid_lut_PIDLookuptest and LOWQ2/EEMC/pid/pid_lut wiring to match the new link-input contracts.
Reviewed changes
Copilot reviewed 28 out of 28 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| src/tests/algorithms_test/pid_lut_PIDLookup.cc | Update test to pass MCRecoParticleLinkCollection into PIDLookup instead of associations. |
| src/global/pid/pid.cc | Rewire MatchToRICHPID factory inputs from associations to links. |
| src/global/pid_lut/pid_lut.cc | Rewire PID LUT chain to use link collections; add LOWQ2 link collectors. |
| src/factories/pid/MatchToRICHPID_factory.h | Switch factory input from MCRecoParticleAssociation to MCRecoParticleLink. |
| src/factories/pid_lut/PIDLookup_factory.h | Switch factory input from associations to links. |
| src/factories/fardetectors/FarDetectorTransportationPreML_factory.h | Switch factory input from associations to links. |
| src/factories/calorimetry/CalorimeterParticleIDPreML_factory.h | Switch factory input from associations to links. |
| src/factories/calorimetry/CalorimeterParticleIDPostML_factory.h | Switch factory input from associations to links. |
| src/detectors/LOWQ2/LOWQ2.cc | Update LOWQ2 wiring to provide track links to far-detector transportation pre-ML. |
| src/detectors/EEMC/EEMC.cc | Update EEMC wiring to provide cluster links to calorimeter PID pre/post-ML. |
| src/algorithms/pid/MatchToRICHPID.h | Update algorithm input contract to accept link collections. |
| src/algorithms/pid/MatchToRICHPID.cc | Use LinkNavigator to propagate truth links/associations for matched RICH PID. |
| src/algorithms/pid_lut/PIDLookup.h | Update algorithm input contract to accept link collections. |
| src/algorithms/pid_lut/PIDLookup.cc | Replace association scan with LinkNavigator iteration; propagate links+associations. |
| src/algorithms/onnx/CalorimeterParticleIDPreML.h | Update pre-ML PID algorithm interface to accept cluster links. |
| src/algorithms/onnx/CalorimeterParticleIDPreML.cc | Replace association scan with LinkNavigator logic for targets/features. |
| src/algorithms/onnx/CalorimeterParticleIDPostML.h | Update post-ML PID algorithm interface to accept cluster links. |
| src/algorithms/onnx/CalorimeterParticleIDPostML.cc | Replace association propagation with LinkNavigator logic. |
| src/algorithms/interfaces/LinkTruthUtils.h | New helper utilities for per-event link navigation and truth-relation emission. |
| src/algorithms/fardetectors/FarDetectorTransportationPreML.h | Update far-detector pre-ML algorithm interface to accept track links. |
| src/algorithms/fardetectors/FarDetectorTransportationPreML.cc | Use LinkNavigator for track truth lookup (first-linked behavior). |
| src/algorithms/fardetectors/FarDetectorLinearTracking.h | Adopt shared link-navigation helper; adjust truth association flow. |
| src/algorithms/fardetectors/FarDetectorLinearTracking.cc | Adopt shared link-navigation helper; guard truth counting and linking. |
| src/algorithms/calorimetry/ImagingClusterReco.h | Remove local primary-particle helper in favor of shared utility. |
| src/algorithms/calorimetry/ImagingClusterReco.cc | Use shared truth helper and stable comparator for deterministic association bookkeeping. |
| src/algorithms/calorimetry/CalorimeterClusterShape.cc | Build links from associations to enable LinkNavigator propagation. |
| src/algorithms/calorimetry/CalorimeterClusterRecoCoG.h | Remove local primary-particle helper in favor of shared utility. |
| src/algorithms/calorimetry/CalorimeterClusterRecoCoG.cc | Use shared truth helper and stable comparator for deterministic association bookkeeping. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 28 out of 28 changed files in this pull request and generated 6 comments.
Comments suppressed due to low confidence (1)
src/algorithms/onnx/CalorimeterParticleIDPostML.cc:36
- Throwing
std::runtime_error("")loses the useful context you already logged. Include a non-empty exception message (e.g., same text as the log line) to make failures actionable in contexts where logs aren’t captured.
if (prediction_tensors->size() != 1) {
error("Expected to find a single tensor, found {}", prediction_tensors->size());
throw std::runtime_error("");
}
This PR applies the include-what-you-use fixes as suggested by https://github.com/eic/EICrecon/actions/runs/30128356954. Please merge this PR into the branch `wdconinc-linknavigator-migration` to resolve failures in PR #2780. Auto-generated by [create-pull-request][1] [1]: https://github.com/peter-evans/create-pull-request Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (3)
src/algorithms/onnx/CalorimeterParticleIDPreML.cc:63
- The
!found_assocbranch warns andcontinues, but the tensor shapes are still set toclusters->size(), so skipping any cluster guarantees a later shape/element-count mismatch (and an exception). Either resize tensors to the number of filled entries or fail fast when a cluster has no linked MCParticle.
if (!found_assoc) {
warning("Can't find association for cluster. Skipping...");
continue;
}
src/algorithms/onnx/CalorimeterParticleIDPreML.h:21
inputClusterLinksis modeled asstd::optional<...>in the algorithm signature, but the implementation treats it as required (throws when null/empty) and the factory always wires it as a mandatoryPodioInput. Make the requirement explicit in the algorithm type to avoid a misleading optional contract.
using CalorimeterParticleIDPreMLAlgorithm = algorithms::Algorithm<
algorithms::Input<edm4eic::ClusterCollection,
std::optional<edm4eic::MCRecoClusterParticleLinkCollection>>,
algorithms::Output<edm4eic::TensorCollection, std::optional<edm4eic::TensorCollection>>>;
src/algorithms/fardetectors/FarDetectorTransportationPreML.cc:85
- When
trackLinksis present but empty (or a given track has no linked MCParticle),target_tensorstill advertises a shape ofinputTracks->size() x 3but no (or fewer) elements are appended. This breaks the stated goal of keeping indexing aligned and can produce inconsistent tensors downstream. Ensure you append 3 values per track whenever the target tensor is created (e.g., NaNs when no link exists).
if (link_nav) {
// Use the first linked MC particle, matching previous first-association behavior.
const auto linked_particles = link_nav->getLinked(track);
if (!linked_particles.empty()) {
auto MCElectronMomentum = linked_particles.front().o.getMomentum() / m_beamE;
target_tensor.addToFloatData(MCElectronMomentum.x);
target_tensor.addToFloatData(MCElectronMomentum.y);
target_tensor.addToFloatData(MCElectronMomentum.z);
}
}
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 28 out of 28 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (4)
src/algorithms/onnx/CalorimeterParticleIDPreML.cc:45
target_tensorsis an optional output (see algorithm signature) but is dereferenced unconditionally. If the optional target output isn’t wired, this will crash. Also,cluster_linksis optional in the input contract but treated as required here; at minimum this should fail fast with a clear error instead of null-dereferencing later.
if (cluster_links == nullptr || cluster_links->empty()) {
error("Cluster links are required for CalorimeterParticleIDPreML");
throw std::runtime_error("Missing inputClusterLinks");
}
edm4eic::MutableTensor feature_tensor = feature_tensors->create();
feature_tensor.addToShape(clusters->size());
feature_tensor.addToShape(11); // p, E/p, azimuthal, polar, 7 shape parameters
feature_tensor.setElementType(1); // 1 - float
edm4eic::MutableTensor target_tensor;
target_tensor = target_tensors->create();
target_tensor.addToShape(clusters->size());
target_tensor.addToShape(2); // is electron, is hadron
target_tensor.setElementType(7); // 7 - int64
src/algorithms/onnx/CalorimeterParticleIDPreML.cc:63
- This path skips clusters without adding any feature/target entries, but the tensor shape was pre-sized to
clusters->size(). That guarantees a shape/data mismatch and an exception later, but with a much less actionable error message.
if (!found_assoc) {
warning("Can't find association for cluster. Skipping...");
continue;
}
src/algorithms/fardetectors/FarDetectorTransportationPreML.cc:60
target_tensoris created whenevertrackLinks != nullptr, even if the collection is empty (or the optional output isn’t wired). That can leavetarget_tensorwith a non-zero shape but no data appended in the loop, producing inconsistent output.
edm4eic::MutableTensor target_tensor;
if (trackLinks != nullptr) {
target_tensor = target_tensors->create();
target_tensor.addToShape(inputTracks->size());
target_tensor.addToShape(3); // px,py,pz
target_tensor.setElementType(1); // 1 - float
}
std::optional<podio::LinkNavigator<edm4eic::MCRecoTrackParticleLinkCollection>> link_nav;
if (trackLinks != nullptr && !trackLinks->empty()) {
link_nav.emplace(*trackLinks);
}
src/algorithms/fardetectors/FarDetectorTransportationPreML.cc:85
- When
link_navis enabled but a particular track has no linked MC particle, no target values are appended for that track. This breaks the implicit 1:1 alignment between feature rows and target rows.
if (link_nav) {
// Use the first linked MC particle, matching previous first-association behavior.
const auto linked_particles = link_nav->getLinked(track);
if (!linked_particles.empty()) {
auto MCElectronMomentum = linked_particles.front().o.getMomentum() / m_beamE;
target_tensor.addToFloatData(MCElectronMomentum.x);
target_tensor.addToFloatData(MCElectronMomentum.y);
target_tensor.addToFloatData(MCElectronMomentum.z);
}
}
No Clang-Tidy warnings found so I assume my comments were addressed
Capybara summary for PR 2780
Last updated 2026-07-27T19:48-04:00 0c9fc20 |
This PR applies the include-what-you-use fixes as suggested by https://github.com/eic/EICrecon/actions/runs/30128356954. Please merge this PR into the branch `wdconinc-linknavigator-migration` to resolve failures in PR #2780. Auto-generated by [create-pull-request][1] [1]: https://github.com/peter-evans/create-pull-request Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
36b622f to
0c9fc20
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (3)
src/algorithms/fardetectors/FarDetectorTransportationPreML.cc:55
target_tensoris created whenevertrackLinksis non-null, even if the collection is empty. In that caselink_navis never constructed and no elements are appended, leaving a tensor with shape based oninputTracks->size()but with no data. Create the target tensor only when the link collection is non-empty (or when the navigator is enabled).
edm4eic::MutableTensor target_tensor;
if (trackLinks != nullptr) {
target_tensor = target_tensors->create();
target_tensor.addToShape(inputTracks->size());
target_tensor.addToShape(3); // px,py,pz
target_tensor.setElementType(1); // 1 - float
}
src/algorithms/onnx/CalorimeterParticleIDPreML.cc:63
- The missing-association branch logs a warning and
continues, but the output tensor shapes are still set toclusters->size(), and a consistency check at the end will always throw if any cluster was skipped. Either handle missing links by emitting placeholder feature/target entries to preserve the fixed shape, or fail fast with a clear error instead of warning+continuing.
if (!found_assoc) {
warning("Can't find association for cluster. Skipping...");
continue;
}
src/algorithms/onnx/CalorimeterParticleIDPreML.h:21
- The algorithm signature declares
inputClusterLinksasstd::optional<...>, but the implementation treats links as required (throws when missing/empty). Consider making the input non-optional to reflect the contract and avoid confusing wiring/configuration.
using CalorimeterParticleIDPreMLAlgorithm = algorithms::Algorithm<
algorithms::Input<edm4eic::ClusterCollection,
std::optional<edm4eic::MCRecoClusterParticleLinkCollection>>,
algorithms::Output<edm4eic::TensorCollection, std::optional<edm4eic::TensorCollection>>>;
|
I think the diffs are again caused by irreproducibility in track-cluster matching. (FYI @veprbl @ruse-traveler) |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 38 out of 38 changed files in this pull request and generated no new comments.
Suppressed comments (3)
src/algorithms/onnx/CalorimeterParticleIDPreML.h:21
CalorimeterParticleIDPreMLdeclares the cluster-link input as optional in the algorithm interface, but the implementation now hard-requires it (throws when null/empty). This makes the API contract misleading for callers and wiring; it should be modeled as a required input type instead ofstd::optional<...>.
using CalorimeterParticleIDPreMLAlgorithm = algorithms::Algorithm<
algorithms::Input<edm4eic::ClusterCollection,
std::optional<edm4eic::MCRecoClusterParticleLinkCollection>>,
algorithms::Output<edm4eic::TensorCollection, std::optional<edm4eic::TensorCollection>>>;
src/algorithms/onnx/CalorimeterParticleIDPreML.cc:64
- This branch says it is "Skipping" clusters with no truth link, but the tensors were already sized to
clusters->size(). Skipping here guarantees a shape/data mismatch (and a later throw) when any cluster lacks a link. Either pre-filter valid clusters before sizing, or fail fast here with a clear error so the tensor shape remains consistent.
if (!found_assoc) {
warning("Can't find association for cluster. Skipping...");
continue;
}
src/algorithms/fardetectors/FarDetectorTransportationPreML.cc:77
target_tensoris shaped toinputTracks->size(), but when a given track has no linked MC particle this loop adds no target entries for that track. That silently produces a tensor whose declared shape doesn't match its data length (and de-aligns feature/target rows). Add a fallback (e.g., 0/0/0 or NaNs) so exactly 3 floats are appended per input track, and avoid relying on.front()if the underlying linked-range type isn't guaranteed to provide it.
if (link_nav.enabled()) {
// Use the first linked MC particle, matching previous first-association behavior.
const auto linked_particles = link_nav.linked(track);
if (!linked_particles.empty()) {
auto MCElectronMomentum = linked_particles.front().o.getMomentum() / m_beamE;
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 38 out of 38 changed files in this pull request and generated 1 comment.
Suppressed comments (1)
src/algorithms/fardetectors/FarDetectorTransportationPreML.cc:77
- When
link_nav.enabled()is true,target_tensoris shaped forinputTracks->size()tracks, but entries are only appended when a given track has at least one linked MC particle. Iflinked_particlesis empty for any track, the target tensor will have fewer thanN*3elements and become inconsistent/misaligned downstream. To keep tensor shape/data aligned, append a placeholder (e.g., zeros) for tracks without a linked particle, or throw if missing links are not allowed.
if (link_nav.enabled()) {
// Use the first linked MC particle, matching previous first-association behavior.
const auto linked_particles = link_nav.linked(track);
if (!linked_particles.empty()) {
auto MCElectronMomentum = linked_particles.front().o.getMomentum() / m_beamE;
| for (edm4eic::Cluster cluster : *clusters) { | ||
| double momentum = NAN; | ||
| { | ||
| if (link_nav.enabled()) { | ||
| // FIXME: use track momentum once matching to tracks becomes available | ||
| edm4eic::MCRecoClusterParticleAssociation best_assoc; | ||
| for (auto assoc : *cluster_assocs) { | ||
| if (assoc.getRec() == cluster) { | ||
| if ((not best_assoc.isAvailable()) || (assoc.getWeight() > best_assoc.getWeight())) { | ||
| best_assoc = assoc; | ||
| } | ||
| edm4hep::MCParticle best_sim; |
|
Missed another primary lookup function in https://github.com/eic/EICrecon/blob/wdconinc-linknavigator-migration/src/algorithms/calorimetry/SimCalorimeterHitProcessor.cc#L55 |
Migrate selected algorithms to link-based truth lookup via podio::LinkNavigator, rewire affected factories and pipeline wiring to consume link collections, and add shared LinkTruthUtils helpers to remove repeated per-event navigator and relation boilerplate. Also harden FarDetector link-absent handling and update PIDLookup algorithm tests for link-input semantics. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This PR applies the include-what-you-use fixes as suggested by https://github.com/eic/EICrecon/actions/runs/30128356954. Please merge this PR into the branch `wdconinc-linknavigator-migration` to resolve failures in PR #2780. Auto-generated by [create-pull-request][1] [1]: https://github.com/peter-evans/create-pull-request Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Migrate selected algorithms to link-based truth lookup via podio::LinkNavigator, rewire affected factories and pipeline wiring to consume link collections, and add shared LinkTruthUtils helpers to remove repeated per-event navigator and relation boilerplate. Also harden FarDetector link-absent handling and update PIDLookup algorithm tests for link-input semantics. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This PR applies the include-what-you-use fixes as suggested by https://github.com/eic/EICrecon/actions/runs/30128356954. Please merge this PR into the branch `wdconinc-linknavigator-migration` to resolve failures in PR #2780. Auto-generated by [create-pull-request][1] [1]: https://github.com/peter-evans/create-pull-request Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Replace the MCRecoClusterParticleAssociation input with MCRecoClusterParticleLink in the algorithm, factory, and all detector wiring. The intermediate local-link-collection workaround in the algorithm is removed; it now constructs an EventLinkNavigator directly from the link input and delegates relation emission to the shared truth::addWeightedRelation helper. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
02b2535 to
4525f29
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 38 out of 38 changed files in this pull request and generated no new comments.
Suppressed comments (3)
src/algorithms/fardetectors/FarDetectorTransportationPreML.cc:79
target_tensoris shaped asinputTracks->size(), but for tracks with no linked MC particle this code adds no target values. That can silently create a tensor whose shape/data length disagree and misalign track indices downstream. If targets are enabled, write 3 values per track (e.g., zeros) even when no link exists, or throw to make the invariant explicit.
const auto linked_particles = link_nav.linked(track);
if (!linked_particles.empty()) {
auto MCElectronMomentum = linked_particles.front().o.getMomentum() / m_beamE;
target_tensor.addToFloatData(MCElectronMomentum.x);
target_tensor.addToFloatData(MCElectronMomentum.y);
src/global/pid_lut/pid_lut.cc:11
pid_lut.ccincludespodio/detail/Link.hand<deque>, but neither is used in this file.podio/detail/*headers are typically internal and can break builds across podio versions; removing these unused includes reduces fragility.
src/algorithms/onnx/CalorimeterParticleIDPreML.cc:63- When
fill_targetsis true but a cluster has no linked MC particle, the code logs "Skipping..." andcontinues. This guaranteesfeature_tensor.floatData_size()will not match the predeclared shape (clusters->size()), so the function will later throw an "Inconsistent output tensor shape" error anyway. Consider failing fast with a clear error here (or alternatively redesigning the tensor shape to only include kept clusters).
if (!found_assoc) {
warning("Can't find association for cluster. Skipping...");
continue;
}
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 38 out of 38 changed files in this pull request and generated 1 comment.
Suppressed comments (4)
src/algorithms/fardetectors/FarDetectorTransportationPreML.cc:77
- This block can produce an inconsistent target tensor: when a track has no linked MC particle, no (px,py,pz) entries are appended even though the tensor shape is set to
inputTracks->size(). It also uses.front()on the linked range, which is brittle if the return type is not a container. Consider iterating the linked range once, taking the first element, and always appending 3 values (e.g., zeros) to keep indexing aligned.
if (link_nav.enabled()) {
// Use the first linked MC particle, matching previous first-association behavior.
const auto linked_particles = link_nav.linked(track);
if (!linked_particles.empty()) {
auto MCElectronMomentum = linked_particles.front().o.getMomentum() / m_beamE;
src/algorithms/onnx/CalorimeterParticleIDPreML.cc:33
feature_tensoralways usesmomentum(andE/p), but when the link collection is missing/empty the current logic leavesmomentumas NaN and silently writes NaNs to the feature tensor. Also,target_tensorsis optional but is dereferenced without a null check. Consider (a) failing fast when links are missing/empty since momentum cannot be computed yet (per the FIXME), and (b) only creating/filling the target tensor whentarget_tensorsis present.
const truth::EventLinkNavigator<edm4eic::MCRecoClusterParticleLinkCollection> link_nav(
cluster_links);
const bool fill_targets = link_nav.enabled();
edm4eic::MutableTensor feature_tensor = feature_tensors->create();
src/algorithms/onnx/CalorimeterParticleIDPreML.cc:49
- Momentum is currently computed only when
fill_targetsis true, but the feature tensor always consumesmomentum(andE/p). IfoutputTargetTensoris disabled (sofill_targetsbecomes false after the suggested guard), this will again emit NaNs into the feature tensor. Momentum should be computed whenever links are available (independent of whether targets are written).
if (fill_targets) {
src/global/pid_lut/pid_lut.cc:11
<podio/detail/Link.h>is an internal PODIO header and appears unused here, and<deque>is also unused. Depending on PODIO internals makes builds more fragile across PODIO upgrades; please remove these unused/internal includes if they are not required.
|
|
||
| edm4eic::MutableTensor target_tensor; | ||
| if (mcAssociation != nullptr) { | ||
| if (link_nav.enabled()) { |
Briefly, what does this PR introduce? Please link to any relevant presentations or discussions.
This refactor removes duplicated truth-link lookup patterns and consolidates
podio::LinkNavigatorusage across reconstruction/PID paths. It adds a shared helper (src/algorithms/interfaces/LinkTruthUtils.h) for per-event navigator setup, weighted relation emission, and primary-particle extraction, then applies it to existing LinkNavigator-heavy algorithms while also migrating several downstream algorithms/factories/wiring from association-input to link-input contracts.Concretely, this includes:
CalorimeterClusterRecoCoGImagingClusterRecoFarDetectorLinearTrackingCalorimeterParticleIDPreML/PostMLPIDLookupMatchToRICHPIDFarDetectorTransportationPreMLpid,pid_lut,LOWQ2,EEMC) including LOWQ2 merged particle-link collectors.pid_lut_PIDLookupto the link-input interface.It also hardens FarDetector behavior when link collections are missing/empty so we avoid invalid navigator dereference and keep association indexing aligned.
What is the urgency of this PR?
What kind of change does this PR introduce?
Please check if any of the following apply
AI usage: Copilot CLI was used to identify duplicated LinkNavigator setup/lookup patterns, implement the helper-based refactor, update wiring and tests, and run build validation.